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Abstract 

Simulations  of  turbulent  reacting  flows  with  chemistry  represented  using 
detailed  kinetic  model  involving  a  large  number  of  species  and  reactions 
are  computationally  expensive.  Here  we  present  a  combined  dimension  re¬ 
duction  and  tabulation  strategy  for  implementing  chemistry  in  large  scale 
parallel  Large-Eddy  Simidation  (LES )/ Probability  Density  Function  (PDF) 
computations  of  turbulent  reacting  flows.  In  this  approach,  the  dimension 
reduction  is  performed  using  the  Rate  Controlled  Constrained- Equilibrium 
(RCCE)  method,  and  tabulation  of  the  reduced  space  is  performed  using  the 
In  Situ  Adaptive  Tabulation  (ISAT)  algorithm.  In  addition,  we  use  x2f_mpi 
-  a  Fortran  library  for  parallel  vector-valued  function  evaluation  (used  with 
ISAT  in  this  context)  -  to  efficiently  redistribute  the  chemistry  workload 
among  the  participating  cores  in  parallel  LES/PDF  computations  to  reduce 
the  overall  wall  clock  time  of  the  simulation.  We  test  three  parallel  strate¬ 
gies  for  redistributing  the  chemistry  workload,  namely  (a)  PLP,  purely  local 
processing;  (b)  URAN,  the  uniform  random  distribution  of  chemistry  com¬ 
putations  among  all  cores  following  an  early  stage  of  PLP;  and  (c)  P-LIRAN, 
a  Partitioned  LIRAN  strategy  that  redistributes  the  workload  within  parti¬ 
tions  or  subsets  of  the  cores.  To  demonstrate  the  efficiency  of  this  combined 
approach,  we  perform  parallel  LES/PDF  computations  (on  1,024  cores)  of 
the  Sandia  Flame  D  with  chemistry  represented  using  a  38-species  C\-C± 
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1  1.  Introduction 

2  Detailed  chemical  mechanisms  of  hydrocarbon  fuels  may  involve  hun- 

3  dreds  or  thousands  of  species  and  thousands  of  reactions  [1,  2],  Incorpo- 

4  rating  directly  such  detailed  chemistry  in  the  combustion  flow  calculations 

5  is  computationally  prohibitive,  even  using  distributed  parallel  computing, 
e  The  current  efforts  aimed  at  reducing  the  computational  cost  of  represent- 
7  ing  chemistry  can  be  placed  under  three  main  categories:  (1)  mechanism 
s  reduction,  the  generation  of  reaction  mechanisms  involving  fewer  species  and 

9  reactions  [3,  4,  5];  (2)  dimension  reduction ,  the  representation  of  chemistry 

10  using  a  reduced  number  of  variables  [6,  7,  8,  9];  and  (3)  tabulation,  the  use  of 

11  storage-retrieval  methods  to  reduce  significantly  the  cost  of  expensive  evalu- 

12  ations  of  the  reaction  mappings  involving  ODE  integrations  [10,  11,  12,  13]. 
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13  Combined  methodologies  have  also  been  developed  to  use  reduced  reaction 

14  mechanisms  or  dimension  reduction  methods  in  conjunction  with  tabulation 

15  [14,  15,  16,  17]. 

16  Since  most  of  the  modern  day  reactive  flow  simulations  are  performed 

17  in  parallel  on  multiple  cores  using  distributed  computing,  in  addition  to  the 
is  aforementioned  techniques,  strategies  are  needed  to  efficiently  distribute  the 
is  chemistry  workload  among  the  participating  cores  to  reduce  the  overall  wall 

20  clock  time  of  the  simulation  [18,  19,  20,  21].  We  recently  demonstrated 

21  scalable  parallel  strategies  implemented  using  x2f_mpi  for  the  efficient  redis- 

22  tribution  of  chemistry  workload  in  large  scale  parallel  Large-Eddy  Simulation 

23  (LES)/Probability  Density  Function  (PDF)  computations  [22], 

24  In  this  paper  we  further  extend  our  LES/PDF  solver  with  the  capa- 

25  bility  of  representing  chemistry  using  our  combined  dimension  reduction 

26  and  tabulation  approach  [15].  In  this  approach,  the  dimension  reduction 

27  is  performed  using  the  Rate  Controlled  Constrained-Equilibrium  (RCCE) 

28  [6,  23,  24]  method  followed  by  tabulation  using  the  In  Situ  Adaptive  Tab- 

29  illation  (ISAT)  [10,  11]  algorithm.  In  [15],  we  tested  our  combined  dirnen- 

30  sion  reduction  and  tabulation  approach  using  the  partially-stirred  reactor  for 

31  methane  and  ethylene  chemistry,  and  the  main  conclusions  drawn  from  this 

32  work  are  that  the  ISAT /RCCE  approach 

33  •  yields  the  same  level  of  accuracy  as  other  reduced  (based  on  the  Quasi- 

34  Steady  State  Assumption,  QSSA)  or  skeletal  mechanisms  with  rela- 

35  tively  fewer  represented  species; 

36  •  yields  significant  speedup  relative  to  using  ISAT  alone  with  the  detailed 

37  mechanism. 

38  Here,  for  the  first  time,  the  ISAT /RCCE  approach  is  being  demonstrated 

39  in  the  context  of  full-scale  LES/PDF  simulations  of  turbulent  reacting  flows 

40  using  realistic  chemistry.  In  this  study,  the  accuracy  and  efficiency  of  this 

41  combined  approach  is  demonstrated  by  performing  full-scale  LES/PDF  sim- 

42  illation  of  the  Sandia  Flame  D  [25]. 

43  The  outline  of  the  remainder  of  the  paper  is  as  follows:  in  Section  2  we 

44  describe  our  combined  LES/PDF  solver;  in  Section  3  we  describe  the  com- 

45  bined  dimension  reduction  and  tabulation  strategy;  in  Section  4  we  briefly 

46  describe  the  parallel  strategies  implemented  using  x2f_mpi  for  redistributing 

47  the  chemistry  workload  in  large  scale  LES/PDF  computations;  in  Section 

48  5  we  present  computational  details  for  simulating  the  Sandia  Flame  D;  in 


3 


49  Section  6  we  present  simulation  results;  and  finally  in  Section  7  we  state  our 
so  conclusions. 

51  2.  Combined  LES/PDF  Solver 

52  In  this  study  we  use  a  combined  LES/PDF  solver  developed  at  Cornell 

53  as  described  in  more  detail  in  [22,  26].  Below  we  mention  some  of  the  key 

54  aspects  of  this  solver. 

55  2.1.  LES  Solver 

56  The  LES  solver  is  based  on  the  Stanford  LES  code  [27,  28].  The  solver 

57  uses  structured  non-uniform  grids;  supports  cylindrical  coordinate  system;  is 

58  second-order  accurate  in  space  and  time;  and  is  parallelized  (using  MPI)  by 

59  domain  decomposition  in  two  dimensions. 

so  2. 2.  PDF  Solver 

61  The  PDF  solver,  HPDF  [26],  has  second-order  accuracy  in  space  and 

62  time;  supports  Cartesian  and  cylindrical  coordinate  systems;  is  parallelized 

63  (using  MPI)  by  domain  decomposition  in  two  dimensions;  and  has  a  general 

64  interface  to  facilitate  coupling  with  existing  LES  (or  RANS)  solvers. 

65  In  the  PDF  approach,  the  thermochemical  composition  of  the  fluid  within 
ee  the  solution  domain  is  represented  by  a  large  number  of  particles.  The  HPDF 

67  solver  has  three  main  components 

68  1.  transport  to  account  for  the  change  in  position  of  a  particle  due  to 

69  advection  in  the  physical  space  (including  a  random-walk  component  to 

70  represent  the  effects  of  subgrid-scale  turbulent  advection  and  molecular 

71  diffusion) ; 

72  2.  mixing :  to  account  for  change  in  the  composition  of  a  particle  due  to 

73  mixing  with  neighboring  particles  (which  models  the  effects  of  molecu- 

74  lar  mixing);  and 

75  3.  reaction :  to  account  for  change  in  the  composition  of  a  particle  due  to 

76  chemical  reaction. 

77  These  components  are  implemented  in  fractional  steps  using  splitting  schemes 

78  [29].  In  this  study,  to  simulate  the  Sandia  Flame  D,  we  use  the  first-order 

79  TMM  splitting  scheme  (which  is  found  to  perform  as  well  as  the  second-order 
so  splitting  scheme  for  jet  flames  [26]).  The  TMM  splitting  scheme  denotes  tak- 
8i  ing  fractional  steps  of  transport,  T;  mixing ,  M;  and  reaction ,  M  in  this  order 
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82  on  each  time-step.  The  Kloeden  and  Platen  (KP)  [30]  stochastic  differential 

83  equation  (SDE)  scheme  is  used  to  integrate  the  transport  equations;  and  the 

84  mixing  is  represented  using  the  modified  Curl  [31]  mixing  model.  The  reac- 

85  tion  fractional  step  is  implemented  using  the  combined  dimension  reduction 
se  and  tabulation  approach  which  we  will  discuss  in  more  detail  in  the  later 

87  sections. 

88  2.3.  Domain  Decomposition 

89  The  LES  computations  are  performed  on  a  structured  non-uniform  grid  in 

90  the  cylindrical  coordinate  system.  We  denote  the  grid  used  for  LES  compu- 

91  tations  by  NxxNrxNg  (in  the  axial,  radial  and  azimuthal  directions).  In  per- 

92  forming  parallel  LES/PDF  computations  (using  the  combined  LES/HPDF 

93  solver)  on  Nc  cores,  the  computational  domain  is  decomposed  into  Nc  sub- 

94  domains  and  each  core  performs  the  computations  of  one  sub-domain.  The 

95  domain  decomposition  is  done  in  the  first  two  principal  directions  x  and  r, 

96  and  is  denoted  by  Dx  x  Dr ,  where  DxDr  =  Nc.  For  instance,  in  this  study  we 

97  perform  LES/PDF  simulations  of  the  Sandia  Flame  D  using  a  non-uniform 

98  LES  grid  of  size  Nx  =  192,  Nr  =  192,  Ng  =  96  on  Nc  =  1,024  cores  using  a 

99  domain  decomposition  with  Dx  =  64  and  Dr  =  16. 

100  3.  Combined  Dimension  Reduction  and  Tabulation 

101  In  this  section  we  briefly  describe  the  combined  dimension  reduction 

102  and  tabulation  approach  used  for  representing  chemistry  using  ISAT/RCCE. 

103  More  detailed  description  can  be  found  in  [15]. 

104  3.1.  Particle  Representation 

105  We  consider  a  reacting  gas-phase  mixture  consisting  of  ns  chemical  species, 

106  composed  of  ne  elements.  We  consider  an  isobaric  system  with  a  fixed  spec- 

107  ihed  pressure  p,  and  so  the  thermochemical  state  of  the  mixture  (at  a  given 

108  position  and  time)  is  completely  characterized  by  the  mixture  enthalpy  h, 

109  and  the  ns-vector  z  of  specific  moles  of  the  species. 

no  In  the  reaction  fractional  step,  a  particle’s  chemical  composition  z  evolves 
in  (at  constant  enthalpy  h)  in  time  according  to  the  following  set  of  ns  coupled 
112  ordinary  differential  equations  (ODEs) 

^  =  S(z( «)),  (1) 
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113  where  S  is  the  n,s-vector  of  chemical  production  rates  determined  by  the 

114  chemical  mechanism  used  to  represent  the  chemistry. 

ns  Given  a  reaction  fractional  time  step  At,  the  reaction  mapping ,  z(A t),  is 
lie  defined  to  be  the  solution  to  Eqn.l  after  time  At  starting  from  the  initial 
ii7  composition  z(0).  The  reaction  mapping  obtained  by  directly  integrating  the 
ns  set  of  ODEs  given  by  Eqn.l  is  referred  to  as  a  direct  evaluation  (DE).  We 
ns  use  DDASAC  [32]  for  performing  ODE  integration. 

120  Owing  to  the  large  cost  of  direct  evaluation  of  reaction  mappings  involv- 

121  ing  large  numbers  of  species,  we  use  a  combined  dimension  reduction  (us- 

122  ing  RCCE)  and  tabulation  (using  IS  AT)  strategy  for  representing  chemistry. 

123  This  combined  methodology  can  be  applied  to  chemical  systems  involving  a 

124  large  number  of  species  (100  to  1000)  by  first  applying  the  dimension  reduc¬ 
es  tion  to  reduce  the  dimensionality  of  the  system  to  say  around  20  (depending 

126  on  the  level  of  accuracy  needed)  and  then  using  ISAT  to  tabulate  the  reaction 

127  mappings  in  the  reduced  dimension. 

128  3.2.  Dimension  Reduction 

129  In  this  section  we  briefly  describe  the  procedure  followed  for  dimension 
no  reduction  in  our  implementation  of  the  RCCE  method;  a  more  detailed  de- 

131  scription  can  be  found  in  [14,  15]. 

132  In  our  implementation  of  RCCE,  to  perform  the  dimension  reduction  we 

133  specify  a  set  of  nrs  represented  (constrained)  species  selected  from  the  full  set 

134  of  ns  species.  Consequently,  we  have  nus  =  ns  —  nrs  unrepresented  species. 

135  The  selection  of  good  represented  species  is  crucial  for  the  accuracy  of 

136  the  RCCE  dimension  reduction  method.  We  have  devised  an  automated 

137  Greedy  Algorithm  with  Local  Improvement  (GALI)  [14,  15]  to  select  good 

138  represented  species  based  on  a  specified  measure  of  dimension  reduction  error. 

139  The  greedy  algorithm  selects  represented  species  in  stages  one-by-one  which 
mo  minimizes  the  specified  measure  of  dimension  reduction  error  [14], 

mi  The  reduced  representation  of  the  species  composition  is  denoted  by  r  = 

142  {zr,  zM,e},  where  z'r  is  an  nrs-vector  of  specific  moles  of  the  represented  species; 

143  and  zM,e  is  an  ne-vector  of  specific  moles  of  the  elements  in  the  unrepresented 

144  species  (for  atom  conservation).  Thus,  r  is  a  vector  of  length  nr  =  nrs  +  ne, 

145  and  the  dimension  of  the  system  is  reduced  from  ns  to  nr.  At  any  time  t , 
we  the  reduced  representation ,  r (t),  is  related  to  the  full  representation,  z(t),  by 

r(t)  =  Brz(t),  (2) 
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147  where  B  is  a  constant  ns  x  nr  matrix  which  can  be  determined  for  a  specified 
ms  set  of  represented  species.  (In  general,  the  reduced  representation  in  RCCE 

149  can  be  a  linear  or  non-linear  function  of  the  full  representation  [33].) 

150  In  the  HPDF  solver,  with  dimension  reduction,  the  particles  carry  only 

151  the  reduced  representation  {r,  h}.  Given  the  reduced  representation  r,  the 

152  temperature  T  and  density  p  are  approximated  using  the  method  described 

153  in  the  Appendix  of  [15].  Later  in  the  Results  (Section  6.1)  we  show  that  this 

154  approximation  method  yields  values  of  temperature  and  density  that  match 

155  closely  with  those  obtained  with  the  full  representation  z. 

156  Given  the  reduced  representation  at  the  beginning  of  the  reaction  frac- 

157  tional  step  r(0),  and  the  reaction  fractional  time  step  At,  the  reduced  reaction 

158  mapping  r(A t)  (at  constant  enthalpy)  is  computed  using  the  following  steps: 

159  1.  species  reconstruction :  given  r(0),  we  compute  the  constrained-equilibrium 

leo  composition  at  constant  enthalpy,  zCi:;(r(0)),  using  CEQ  [34]; 

lei  2.  reaction  mapping-,  starting  from  zC£:(r(0)),  we  solve  the  full  system  of 

162  ns  ODEs  Eqn.l  to  obtain  the  reaction  mapping,  z(A t); 

163  3.  reduction-,  we  obtain  the  reduced  reaction  mapping  as,  r( At)  =  B7z( At). 

164  The  above  steps  of  course  make  the  computation  of  the  reaction  mapping 

165  even  more  expensive  than  directly  solving  the  full  set  of  ODEs  Eqn.l,  due 
lee  to  the  additional  species  reconstruction  and  reduction  steps.  However,  when 
i67  IS  AT  is  used  in  conjunction  with  dimension  reduction,  the  computational 
lea  cost  is  reduced  significantly  as  explained  in  the  next  section. 

169  A  more  efficient  way  of  obtaining  the  reduced  reaction  mapping,  r(A t), 

170  is  to  directly  solve  a  reduced  system  of  nr  ODEs  for  the  constraints,  r,  or  for 

171  the  constraint-potentials  (as  is  done  in  the  classical  RCCE  approach  [35,  36]). 

172  We  are  currently  working  on  implementing  this  method  which  should  give 

173  a  further  improvement  in  performance.  Nevertheless,  even  with  our  current 

174  implementation  of  RCCE,  we  achieve  significant  reduction  in  computational 

175  cost  relative  to  the  detailed  chemistry  calculation  as  shown  in  this  work. 

to  3.3.  Tabidation 

177  We  use  in  situ  adaptive  tabulation  (ISAT)  [10]  for  tabulating  the  reac¬ 
ts  tion  mappings.  The  ISAT  algorithm  has  been  successfully  applied  in  many 
179  combustion  chemistry  calculations  involving  up  to  ns  <  50  species  [11,  15]. 
iso  However,  with  chemistry  involving  more  than  50  species,  the  direct  use  of 

181  ISAT  may  not  be  very  efficient,  due  to  the  large  table  size  and  search  times 

182  [15], 


7 


183  Hence,  for  chemistry  involving  more  than  say  ns  >  30  species,  we  use 

184  the  RCCE  dimension  reduction  method  to  represent  the  chemistry  using  a 
las  reduced  representation  involving  fewer  nr  variables.  Note,  for  very  large 
we  mechanisms  involving  thousands  of  species,  the  direct  use  of  RCCE/GAL1 
i87  may  still  result  in  nr  3>  30  to  achieve  an  acceptable  level  of  accuracy.  In  such 
las  cases  it  will  be  more  efficient  to  use  ISAT/RCCE  with  a  skeletal  mechanism 

189  (based  on  the  detailed  mechanism)  involving  hundreds  of  species. 

190  We  use  ISAT  to  tabulate  the  reduced  reaction  mapping  in  the  reduced 

191  dimension  nr ,  which  reduces  significantly  the  overall  computational  cost  be- 

192  cause 


193 

194 

195 

196 

197 

198 

199 

200 
201 
202 


1.  the  exact  reduced  reaction  mapping  is  computed  (using  the  steps  listed 
in  the  previous  section)  only  for  a  small  fraction  of  particles  (typically 
less  than  1%);  and  for  the  majority  of  the  particles  a  linear  approxima¬ 
tion  to  the  reduced  reaction  mapping  is  obtained  using  the  tabulated 
data; 

2.  since  the  tabulation  is  performed  in  a  reduced  dimension,  nr ,  the  ISAT 
table  size  is  reduced,  which  in  turn  reduces  the  table  search  and  retrieve 
times;  and 

3.  since  the  particle  compositions  are  also  stored  in  a  reduced  dimension, 
fetching  particles  from  the  memory  is  faster. 


203  Consequently,  the  combined  dimension  reduction  and  tabulation  approach 

204  using  ISAT /RCCE  is  found  to  give  an  additional  speedup  by  a  factor  of  0(10) 

205  relative  to  using  ISAT  alone  with  the  full  representation  for  tests  performed 

206  using  the  111-species  C1-C4  USC  Mech  II  detailed  mechanism  [15].  A  more 

207  detailed  description  of  our  combined  dimension  reduction  and  tabulation  ap- 

208  proach  is  provided  in  [15]. 


209  4.  Parallel  Strategies  for  Implementing  Chemistry 

210  In  performing  parallel  LES/PDF  computations  on  multiple  cores  using 

211  our  LES /HPDF  solver  with  chemistry  represented  using  the  combined  dimen- 

212  sion  reduction  and  tabulation  approach,  each  core  has  its  own  ISAT  table 

213  for  tabulating  the  chemistry.  On  the  reaction  fractional  step,  the  reaction 

214  mappings  for  all  the  particles  in  the  computational  domain  need  to  be  eval- 

215  uated.  However,  the  chemical  reactivity  is  in  general  not  uniform  across  the 

216  entire  domain.  For  example,  in  simulation  of  jet  flames,  the  sub-domains  in 

217  the  flame  front  are  chemically  more  reactive  than  sub-domains  in  the  outer 
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coflow/air.  Thus,  a  direct  call  to  ISAT  on  each  core  at  the  reaction  fractional 
step  can  create  load  imbalance  among  the  cores,  leading  to  increase  in  the 
overall  simulation  wall  clock  time.  Hence,  at  the  reaction  fractional  step,  we 
use  parallel  strategies  implemented  using  x2f_mpi  to  redistribute  the  parti¬ 
cles  among  the  participating  cores  for  reaction  mapping  evaluation,  thereby 
achieving  a  better  load  balance  and  reducing  the  overall  simulation  wall  clock 
time. 

In  [22],  we  presented  three  parallel  strategies,  denoted  by  PLP,  UR  AN 
and  P-URAN,  for  redistributing  the  chemistry  workload.  We  give  a  brief 
description  of  these  strategies  below. 

1.  Purely  Local  Processing  (PLP): 

In  this  strategy,  the  reaction  mapping  of  all  the  particles  on  a  core 
is  evaluated  using  the  local  ISAT  table  without  any  message  passing  or 
load  redistribution.  This  in  some  sense  is  the  same  as  invoking  ISAT 
directly  from  HPDF  on  each  core  without  using  the  x2f_mpi  interface. 
This  strategy  thus  leads  to  significant  load  imbalance. 

2.  Uniform  Random  (URAN): 

This  strategy  is  the  extreme  opposite  of  the  PLP  strategy  and  aims 
at  achieving  statistically  ideal  load  balancing  by  evenly  distributing 
the  chemistry  workload  among  all  the  participating  cores.  The  strategy 
involves  one  initial  step  of  PLP  to  initialize  the  local  ISAT  tables.  In  the 
subsequent  steps,  on  each  core,  first  an  attempt  is  made  to  retrieve  the 
reaction  mapping  of  particles  from  the  local  ISAT  table  (also  referred  to 
as  a  “quick  try” ).  Following  this,  there  is  a  uniform  random  distribution 
of  all  the  unresolved  particles  (for  which  “quick  try”  failed)  to  all  the 
cores.  This  strategy  thus  ensures  that  the  workload  is  evenly  balanced 
over  all  the  cores,  however,  it  also  results  in  a  large  amount  of  all-to-all 
message  passing. 

3.  Partitioned  Uniform  Random  (P-URAN): 

This  strategy  aims  at  achieving  a  balance  between  communication 
cost  and  load  imbalance  by  using  the  PLP  and  URAN  strategies  over 
smaller  partitions  of  cores.  The  P-URAN  strategy  works  in  two  stages: 
in  stage  1,  for  a  specified  duration  of  time  r  (hours)  the  PLP  strategy  is 
used  to  resolve  particles;  then  in  stage  2,  for  the  remainder  of  the  time, 
the  participating  cores  are  partitioned  into  smaller  groups  of  k  cores, 
and  within  each  partition  the  LIRAN  strategy  is  used  to  uniformly 
distribute  the  chemistry  workload  among  the  cores  in  that  partition. 
We  use  the  notation  P-URAN  [r,  k]  to  describe  the  P-URAN  strategy. 
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256  In  [22],  based  on  LES/PDF  simulations  of  Sandia  Flame  D  using  IS  AT 

257  alone  we  showed  that  among  the  aforementioned  three  strategies,  the  P- 

258  URAN  strategy  yields  the  lowest  wall  clock  time.  We  also  showed  that  the 

259  P-URAN  strategy  shows  good  scaling  up  to  9,000  cores.  In  this  work  we 

260  use  these  strategies  in  conjunction  with  combined  dimension  reduction  and 

261  tabulation  to  compare  their  relative  performance.  Here  we  focus  more  on 

262  the  gains  achieved  using  the  combined  dimension  reduction  and  tabulation 

263  approach  and  show  that  the  simulation  wall  clock  time  can  be  further  reduced 

264  using  our  combined  ISAT /RCCE  approach  without  losing  too  much  accuracy. 

265  5.  LES/PDF  Simulation  of  Sandia  Flame  D 

266  To  test  the  chemistry  implementation  we  perform  LES/PDF  simulations 

267  of  the  Sandia  Flame  D. 

268  5.1.  Sandia  Flame  D 

269  The  Sandia  Flame  D  is  a  piloted  CH4/ Air  jet  flame  operating  at  a  jet 

270  Reynolds  number,  Re  =  22,400.  All  the  details  about  this  flame  and  the 

271  burner  geometry  can  be  found  at  [25].  Here  we  mention  only  some  of  the 

272  important  aspects  of  this  flame. 

273  The  jet  fluid  consists  of  25%  Cl H4  and  75%  air  by  volume.  The  jet  flows 

274  in  at  49.6  m/s  velocity  at  294  K  temperature  and  0.993  atm  pressure.  The 

275  jet  diameter,  D  =  7.2  mm.  The  pilot  is  a  lean  (equivalence  ratio,  =  0.77) 

276  mixture  of  C2H2,  H2,  air,  C02l  and  N2  with  the  same  nominal  enthalpy  and 

277  equilibrium  composition  as  that  of  CH4/ Air  at  this  equivalence  ratio.  The 

278  pilot  velocity  is  11.4  m/s.  The  coflow  is  air  flowing  in  at  0.9  m/s  at  291  K 

279  and  0.993  atm. 

280  5.2.  Computational  Details 

281  We  perform  LES/PDF  simulation  of  the  Sandia  Flame  D  using  the  cou- 

282  pled  LES/HPDF  solver.  The  simulation  is  performed  in  a  cylindrical  co- 

283  ordinate  system.  A  computational  domain  of  80Dx30Dx2tt  is  used  in  the 

284  axial,  radial  and  azimuthal  directions,  respectively.  A  non-uniform  struc- 

285  tured  grid  of  size  192x192x96  (in  the  axial,  radial  and  azimuthal  directions, 

286  respectively)  is  used  for  the  LES  solver.  In  the  HPDF  solver,  the  number  of 

287  particles  per  LES  cell  ( Npc )  used  is  Npc  =  40.  With  a  total  of  192x192x96 

288  ~  3.5  x  106  LES  cells,  an  overall  140  x  106  particles  are  used  in  the  computa- 

289  tional  domain.  The  simulations  are  performed  on  1,024  cores  using  a  domain 
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290  decomposition  of  64x16  in  the  axial  and  radial  directions,  respectively.  All 

291  the  simulations  are  performed  on  the  Texas  Advanced  Computing  Center 

292  (TACC)  Ranger  cluster. 

293  The  chemistry  is  represented  using  the  combined  dimension  reduction 

294  (using  RCCE)  and  tabulation  (using  ISAT)  approach  with  a  Cj-C'4  skeletal 

295  mechanism  [37]  involving  ns  =  38  species  composed  of  ne  =  5  elements.  This 

296  mechanism  is  developed  especially  for  ethylene  combustion,  but  is  also  appli- 

297  cable  to  methane  flames.  In  the  future,  we  want  to  apply  the  methodology 

298  developed  here  to  study  ethylene  combustion. 

299  The  RCCE  dimension  reduction  is  performed  by  specifying  nrs  =  10  rep- 

300  resented  species  (which  is  found  to  be  a  good  number  of  represented  species 

301  to  achieve  less  than  2%  dimension  reduction  error  based  on  our  previous  tests 

302  with  chemical  mechanisms  involving  around  30  species  [14,  15,  16,  17]),  and 

303  so  the  reduced  representation  has  a  dimension  nr  =  nrs  +  ne  =  15.  This  di- 

304  mension  reduction  from  ns  =  38  to  nr  =  15  results  in  a  60%  reduction  in  the 

305  storage  needed  for  particle  composition  and  an  84%  reduction  in  the  storage 

306  per  ISAT  table  entry.  In  this  preliminary  study,  we  specify  the  represented 

307  species  manually  (to  include  the  major  species  of  interest  for  which  statistics 

308  had  already  been  collected  in  some  of  our  previous  LES/PDF  simulations  and 

309  for  which  experimental  data  is  available).  However,  in  future  studies  with 

310  bigger  mechanisms  we  will  use  GALI  [15]  to  select  the  represented  species. 

311  In  this  work,  we  use  the  following  10  species  as  the  represented  species:  CH±, 

312  02,  CO2,  H20,  CO,  H-2,  OH,  H,  O  and  H02. 

313  A  fixed  ISAT  error  tolerance,  etoi  =  10-4,  is  used  in  this  study.  At 

314  this  error  tolerance,  the  ISAT  tabulation  error  relative  to  direct  evaluation 

315  (as  defined  in  [15])  is  found  to  be  less  than  3%.  In  addition,  we  specify  a 

316  maximum  ISAT  table  size  of  1000  MB  per  core.  In  simulations  with  the 

317  38-species  full  representation,  some  ISAT  tables  become  completely  filled. 

318  However  in  simulations  with  the  combined  ISAT /RCCE  approach  with  10 

319  represented  species,  none  of  the  ISAT  tables  have  a  size  of  more  than  200 

320  MB. 

321  6.  Results 

322  In  this  section  we  compare  the  computational  time  and  statistics  of  ther- 

323  mochemical  quantities  obtained  using  the  combined  dimension  reduction  and 

324  tabulation  approach  with  10  represented  species  relative  to  using  tabulation 

325  alone  with  the  38-species  Cj-C'4  skeletal  mechanism. 
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In  order  to  make  the  comparisons,  we  perform  separate  LES/PDF  sim- 

327  ulations  of  the  Sandia  Flame  D  on  1,024  cores  with  chemistry  represented 

328  using  the  following  two  methods 

329  1.  ISAT:  tabulation  alone  (no  dimension  reduction)  with  the  38-species 

330  full  representation;  and 

331  2.  ISAT /RCCE:  combined  dimension  reduction  and  tabulation  with  a  re- 

332  duced  representation  involving  10  represented  species  (specified  in  the 

333  previous  section)  and  5  elements. 

334  In  each  case,  we  perform  LES/PDF  simulation  to  reach  a  statistically  sta- 

335  tionary  state.  We  then  collect  statistics  for  thermochemical  quantities  like 

336  temperature,  density,  and  species  mass  fractions  time-averaged  over  10,000 

337  time  steps.  In  addition,  in  each  case  we  perform  simulations  for  2,000  time 

338  steps  using  the  three  parallel  strategies  PLP,  URAN  and  P-URAN  to  com- 

339  pare  their  relative  performance.  These  simulations  start  from  the  statistically 

340  stationary  state  with  empty  ISAT  tables. 

341  6.1.  Comparison  of  Statistics 

342  In  this  section  we  compare  the  radial  profiles  of  mean  and  standard  devi- 

343  ation  statistics  of  thermochemical  quantities  obtained  from  the  PDF  particle 

344  data  from  the  LES /PDF  simulation  using  ISAT  alone  and  using  the  combined 

345  ISAT/RCCE  approach. 

346  The  radial  statistics  are  azimuthally-averaged  at  each  time  step,  and  are 

347  also  time-averaged  over  10,000  time  steps  after  reaching  the  statistically 

348  stationary  state.  For  a  quantity  £,  we  denote  the  density-weighted  mean 

349  statistics  by  (£),  and  the  standard  deviation  by  (£")  which  is  defined  as 

»  <f">  =  VW)  -  W ■  ' 

351  In  Fig.l  and  Fig. 2,  we  show  respectively  the  radial  profiles  of  mean  and 

352  standard  deviation  of  temperature  T,  density  p,  and  mass  fraction  of  species 

353  CH4,  O2,  CO2,  H20,  CO ,  H2,  OH  at  axial  locations  x/D  =  15,  30,  45,  60 

354  obtained  from  (i)  an  LES/PDF  simulation  using  ISAT  alone  with  the  38- 

355  species  full  representation;  (ii)  an  LES/PDF  simulation  using  ISAT/RCCE 

356  with  10  represented  species;  and  (iii)  experimentally  measured  statistics  [25] 

357  (for  reference). 

358  We  notice  that  the  statistics  obtained  with  ISAT/RCCE  using  10  rep- 

359  resented  species  match  very  closely  with  the  statistics  obtained  using  ISAT 

360  alone  with  the  38-species  full  representation.  To  quantify  the  difference  be- 

361  tween  the  statistics  obtained  using  ISAT/RCCE  and  ISAT  alone,  for  each 
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362  quantity  £  (mean  or  standard  deviation),  we  compute  the  normalized  root- 

363  mean-square  difference  (RMSD)  denoted  by  e(£)  as  follows 

\c  -  £/l 

e(£)  =  L  ,  irms ,  (3) 

Sre/ 

364  where  £r  and  denote  respectively  the  quantities  obtained  using  the  re- 

365  duced  representation  with  ISAT/RCCE  and  the  full  representation  with 

366  ISAT  alone;  and  the  operator  []rms  denotes  the  RMSD  computed  over  all  the 

367  radial  locations  at  all  the  considered  axial  locations  x/D  =  15,  30,  45,  60. 

368  Here  £rey  is  a  reference  value  used  for  normalization,  which  is  taken  to  be 

369  1000  K  for  temperature  and  1  kg/m3  for  density.  For  the  species  mass  frac- 

370  tions,  we  take  £re/  to  be  the  maximum  value  of  the  mean  statistics  obtained 

371  for  that  species,  max((£)^). 

372  The  reference  value  and  the  normalized  RMSD  computed  using  the  above 

373  expression  Eqn.3  for  all  the  quantities  of  interest  is  summarized  in  Table  1. 

374  We  notice  that  the  normalized  RMSD  in  the  mean  and  standard  deviation 

375  statistics  is  less  than  3%  (i.e.  30  K)  for  temperature;  less  than  2%  (i.e.  0.02 

376  kg/m3)  for  density;  less  than  2.5%  for  species  mass  fractions  of  major  species 

377  CH±,  O2,  CO2,  H2O ;  and  less  than  8%  for  species  mass  fractions  of  minor 

378  species  CO,  H2,  OH.  In  summary,  these  results  show  that  the  combined 

379  ISAT /RCCE  approach  shows  good  error  control  and  the  predicted  statistics 

380  are  well  within  acceptable  level  of  accuracy  (relative  to  using  ISAT  alone  with 

381  the  full  representation)  for  most  engineering  applications.  These  results  also 

382  show  that  the  density  and  temperature  approximation  method  used  with  the 

383  reduced  representation  in  ISAT /RCCE  [15]  yields  values  that  match  closely 

384  with  those  obtained  with  the  full  representation.  A  more  careful  selection  of 

385  represented  species  using  GALI  [15]  should  help  further  reduce  the  differences 

386  between  the  reduced  and  full  descriptions. 

387  The  experimentally  measured  statistics  are  qualitatively  well  captured 

388  by  the  LES/PDF  simulation,  yet  quantitatively  we  notice  that  some  of  the 

389  statistics  differ  by  around  20%.  The  discrepancies  between  the  LES/PDF 

390  simulation  statistics  and  experimentally  measured  statistics  can  be  attributed 

391  to  (i)  numerical  and  statistical  errors  in  the  simulation;  (ii)  experimental 

392  measurement  errors;  and  (iii)  errors  in  the  chemical  kinetic  models.  However, 

393  study  of  these  errors  is  not  the  primary  focus  of  this  work.  Similar  level  of 

394  agreement  between  the  simulated  and  experimentally  measured  statistics  is 

395  found  in  some  of  the  previous  studies  of  Sandia  Flame  D  [38,  39,  40,  41], 
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396  6.2.  Computational  Performance 

397  In  this  section  we  compare  the  wall  clock  time  required  to  perform  LES /PDF 

398  simulation  of  Sandia  Flame  D  for  2,000  time  steps  using  the  combined  ISAT /RCCE 

399  approach  relative  to  using  ISAT  alone.  In  addition  we  compare  the  relative 

400  performance  of  the  PLP,  URAN  and  P-LIRAN  parallel  strategies.  In  each 

401  case,  the  LES /PDF  simulation  is  started  from  a  fixed  statistically  stationary 

402  state  with  empty  ISAT  tables.  We  measure  a  moderate  ISAT  build  time  (see 

403  [15])  of  about  1  hour  in  these  simulations,  i.e.,  after  1  hour  of  simulation, 

404  most  of  the  particles  are  resolved  by  ISAT  retrieves. 

405  In  Fig.3,  the  bottom  three  bars  show  the  wall  clock  time  taken  to  perform 

406  2,000  simulation  time  steps  using  the  combined  ISAT /RCCE  approach  with 

407  10  represented  species  with  the  PLP,  LIRAN  and  P-LIRAN  [0.2h, 32]  parallel 

408  strategies.  In  each  case,  we  also  show  the  breakdown  of  time  spent  in  LES, 

409  HPDF  (outside  reaction),  Reaction  (including  x2f_mpi  communication),  and 

410  Waiting  (average  idle  time)  as  defined  in  [22],  We  see  that  the  P-LIRAN 

411  strategy  yields  the  lowest  wall  clock  time  among  the  three  strategies.  The 

412  Waiting  time  (average  idle  time),  which  is  indicative  of  the  load  imbalance  is 

413  maximum  for  PLP,  minimum  for  LIRAN  and  moderate  for  P-URAN  (as  also 

414  seen  in  our  previous  studies  [22]). 

415  The  LES/PDF  simulation  of  Sandia  Flame  D  using  ISAT  alone  with  the 

416  38-species  full  representation  is  performed  using  the  PLP,  LIRAN  and  P- 

417  URAN  strategies  [22],  Among  these  the  P-LIRAN  strategy  again  yields  the 

418  lowest  wall  clock  time.  In  Fig.3,  for  comparison,  the  top  bar  shows  the  wall 

419  clock  time  for  2,000  time  steps  using  the  38-species  full  representation  with 

420  the  P-LIRAN[0.2h,32]  strategy. 

421  We  notice  that  relative  to  the  simulation  using  ISAT  alone  with  the  38- 

422  species  full  representation,  the  combined  ISAT /RCCE  approach  with  10  rep- 

423  resented  species  using  the  P-LIRAN  strategy 

424  1.  yields  more  than  40%  reduction  in  HPDF  time  (outside  Reaction).  This 

425  is  because  with  dimension  reduction,  the  particles  in  PDF  simulation 

426  carry  only  the  reduced  representation  (in  this  case  involving  15  vari- 

427  ables).  As  a  result,  a)  particles  require  60%  less  storage,  which  in  turn 

428  reduces  the  particle  communication  cost;  and  b)  less  time  is  required 

429  for  collecting  species  (LES  cell  mean)  statistics. 

430  2.  reduces  the  Reaction  time  by  over  40%  due  to  smaller  ISAT  table  sizes 

431  and  faster  retrieve  times  (statistics  given  in  Table  2  and  explained  be- 

432  low) ;  and 
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3.  consequently,  reduces  the  overall  wall  clock  time  of  the  simulation  by 

434  more  than  40%. 

435  In  Table  2,  we  list  the  ISAT  statistics  collected  from  the  simulations 

436  with  the  P-URAN  strategy  using  a)  ISAT  alone  with  the  38-species  full 

437  representation;  and  b)  ISAT/RCCE  with  10  represented  species.  We  see  that 

438  in  both  the  cases,  over  99.9%  of  the  queries  result  in  retrieves  which  shows 

439  the  high  efficiency  of  ISAT  tabulation.  Compared  to  ISAT /RCCE  with  10 

440  represented  species,  the  simulation  with  38-species  full-representation  results 

441  in  almost  twice  the  number  of  adds,  and  also  results  in  some  direct  evaluations 

442  because  some  of  the  tables  get  completely  filled.  The  average  retrieve  time 

443  with  ISAT /RCCE  is  only  3  ps  compared  to  8  p,s  with  ISAT  alone. 

444  7.  Conclusions 

445  We  have  successfully  extended  our  LES/PDF  solver  with  the  capability 

446  of  performing  turbulent  combustion  calculations  with  realistic  combustion 

447  chemistry,  wherein  the  chemistry  in  the  PDF  solver  is  represented  using  the 

448  combined  dimension  reduction  (using  RCCE)  and  tabulation  (using  ISAT) 

449  approach.  The  chemistry  workload  is  efficiently  redistributed  using  the  P- 

450  URAN  strategy  implemented  using  the  x2f^mpi  library.  We  have  shown  that 

451  for  performing  LES/PDF  simulation  of  Sandia  Flame  D,  relative  to  using 

452  ISAT  alone  with  the  38-species  full  representation,  the  ISAT /RCCE  approach 

453  with  10  represented  species  (i)  yields  acceptable  level  of  accuracy  in  mean 

454  and  standard  deviation  statistics  of  major  thermochemical  quantities  of  in- 

455  terest  like  temperature,  density  and  species  mass  fractions;  and  (ii)  reduces 

456  the  overall  simulation  wall  clock  time  by  more  than  40%  with  the  P-URAN 

457  strategy. 

458  8.  Acknowledgments 

459  V.H.’s  work  on  dimension  reduction  methodologies  is  supported  by  Office 

460  of  Energy  Research,  Office  of  Basic  Energy  Sciences,  Chemical  Sciences,  Geo- 

461  sciences  and  Biosciences  Division  of  the  US  Department  of  Energy  (DOE) 

462  under  Grant  No.  DE-FG02-90ER.  The  work  of  V.H.  and  S.R.L.  on  the  ethy- 

463  lcne  mechanism  is  supported  by  Grant  No.  FA9550-09-1-0611  funded  by  the 

464  National  Center  for  Hypersonic  Combined  Cycle  Propulsion,  sponsored  by 

465  the  AFOSR  and  NASA  ARMD.  S.R.L. ’s  initial  work  on  this  project  was 


15 


466  supported  by  NASA  Grant  No.  NNX08AB36A.  This  research  was  also  sup- 

467  ported  in  part  by  the  National  Science  Foundation  through  TeraGrid  re- 

468  sources  provided  by  the  Texas  Advanced  Computing  Center  under  Grant 

469  No.  TG-CTS090020.  S.B.P.  has  a  financial  interest  in  Ithaca  Combustion 

470  Enterprise,  LLC.,  which  has  licensed  the  software  ISAT-CK  and  CEQ  used 

471  in  this  work. 

References 

[1]  C.  K.  Westbrook,  W.  J.  Pitz,  O.  Herbinet,  H.  J.  Curran,  E.  J.  Silkc, 
Combust.  Flame  156  (2009)  181-199. 

[2]  S.  Sarathy,  C.  Westbrook,  M.  Mehl,  W.  Pitz,  C.  Togbe,  P.  Dagaut, 
H.  Wang,  M.  Oehlschlaeger,  U.  Niemann,  K.  Seshadri,  P.  Veloo,  C.  Ji, 
F.  Egolfopoulos,  T.  Lu,  Combustion  and  Flame  158(12)  (2011)  2338- 
2357. 

[3]  T.  Lu,  C.  K.  Law,  Proc.  Combust.  Inst.  30  (2005)  1333-1341. 

[4]  P.  Pepiot-Desjardins,  H.  Pitsch,  Combustion  and  Flame  154  (2008)  67- 
81. 

[5]  T.  Nagy,  T.  Turanyi,  Combustion  and  Flame  156  (2009)  417-428. 

[6]  J.  C.  Keck,  D.  Gillespie,  Combust.  Flame  17  (1971)  237-241. 

[7]  S.  H.  Lam,  D.  A.  Goussis,  International  Journal  of  Chemical  Kinetics 
26  (1994)  461-486. 

[8]  LI.  Maas,  S.  B.  Pope,  Combust.  Flame  88  (1992)  239-264. 

[9]  Z.  Ren,  S.  B.  Pope,  A.  Vladimirsky,  J.  M.  Guckenheimer,  J.  Chem. 
Phys.  124  (2006)  Art.  No.  114111. 

[10]  S.  B.  Pope,  Combust.  Theory  Model.  1  (1997)  41-63. 

[11]  L.  Lu,  S.  B.  Pope,  Journal  of  Computational  Physics  228(2)  (2009)  361— 
386. 

[12]  S.  R.  Tonse,  N.  W.  Moriarity,  N.  J.  Brown,  M.  Frenklach,  Israel  J.  Chem. 
39  (1999)  97-106. 


16 


[13]  T.  Turanyi,  Computers  Chem.  18  (1994)  45-54. 

[14]  V.  Hiremath,  Z.  Ren,  S.  B.  Pope,  Combustion  Theory  and  Modelling 
14(5)  (2010)  619-652. 

[15]  V.  Hiremath,  Z.  Ren,  S.  B.  Pope,  Combustion  and  Flame  158(11)  (2011) 
2113-2127. 

[16]  Z.  Ren,  G.  M.  Goldin,  V.  Hiremath,  S.  B.  Pope,  Combustion  Theory 
and  Modelling  15(6)  (2011)  827-848. 

[17]  Z.  Ren,  G.  M.  Goldin,  V.  Hiremath,  S.  B.  Pope,  Combustion  and  Flame 
(submitted)  (2011). 

[18]  L.  Lu,  Z.  Ren,  V.  Raman,  S.  B.  Pope,  H.  Pitsch,  Proceedings  of  the 
CTR  Summer  Program  (2004)  283-294. 

[19]  L.  Lu,  Z.  Ren,  S.  R.  Lantz,  V.  Raman,  S.  B.  Pope,  H.  Pitsch,  Proceedings 
of  the  4th  Joint  Meeting  of  US  Sections  of  the  Combustion  Institute, 
Philadelphia,  PA  (2005). 

[20]  L.  Lu,  S.  R.  Lantz,  Z.  Ren,  S.  B.  Pope,  Journal  of  Computational  Physics 
228(15)  (2009)  5490-5525. 

[21]  Y.  Shi,  W.  H.  Green,  H.-W.  Wong,  O.  O.  Oluwole,  Combustion  and 
Flame  158(5)  (2011)  836-847. 

[22]  V.  Hiremath,  S.  R.  Lantz,  H.  Wang,  S.  B.  Pope,  Combustion  and  Flame 
(accepted)  (2012)  doi:10.1016/j.combustflame.2012.04.013. 

[23]  J.  C.  Keck,  Prog.  Energy  Combust.  Sci.  16  (1990)  125-154. 

[24]  W.  Jones,  S.  Rigopoulos,  Combustion  and  Flame  142(3)  (2005)  223  - 
234. 

[25]  R.  S.  Barlow,  J.  H.  Frank,  Proc.  Combust.  Inst.  27  (1998)  1087-1095. 
http : / /www . sandia . gov/TNF/DataArch/FlameD . html. 

[26]  H.  Wang,  S.  B.  Pope,  Proceedings  of  the  Combustion  Institute  33  (2011) 
1319-1330. 

[27]  C.  D.  Pierce,  Progress-variable  approach  for  Large-Eddy  Simulation  of 
turbulent  combustion,  Ph.D.  thesis,  Stanford  University,  2001. 


17 


[28]  C.  D.  Pierce,  P.  Moin,  Journal  of  Fluid  Mechanics  504  (2004)  73-97. 

[29]  H.  Wang,  P.  P.  Popov,  S.  B.  Pope,  Journal  of  Computational  Physics 
229  (2010)  1852-1878. 

[30]  P.  Kloeden,  E.  Platen,  Springer  Verlag  Berlin  (1992). 

[31]  J.  Janicka,  W.  Kolbe,  W.  Kollmann,  Journal  of  Non- Equilibrium  Ther¬ 
modynamics  4  (1970)  47-66. 

[32]  M.  Caracotsios,  W.  E.  Stewart,  Computers  &  Chemical  Engineering  9 
(1985)  359-365. 

[33]  G.  P.  Beretta,  J.  C.  Keck,  M.  Janbozorgi,  H.  Metghalchi,  Entropy  14(2) 
(2012)  92-130. 

[34]  S.  B.  Pope,  Combustion  and  Flame  139  (2004)  222-226. 

[35]  M.  Janbozorgi,  H.  Metghalchi,  Int.  J.  of  Thermodynamics  12  (2009) 
44-50. 

[36]  M.  Janbozorgi,  S.  Ugarte,  H.  Metghalchi,  J.  C.  Keck,  Combustion  and 
Flame  156  (2009)  1871-1885. 

[37]  G.  E.  Esposito,  H.  K.  Chelliah,  Combustion  and  Flame  158(3)  (2011) 
477-489. 

[38]  M.  Sheikhi,  T.  Drozda,  P.  Givi,  F.  Jaberi,  S.  Pope,  Proceedings  of  the 
Combustion  Institute  30  (2005)  549-556. 

[39]  M.  Ihme,  H.  Pitsch,  Combustion  and  Flame  155  (2008)  90-107. 

[40]  R.  Mustata,  L.  Valio,  C.  Jimnez,  W.  Jones,  S.  Bondi,  Combustion  and 
Flame  145  (2006)  88-104. 

[41]  M.  B.  Nik,  S.  L.  Yilrnaz,  P.  Givi,  M.  R.  H.  Sheikhi,  S.  B.  Pope,  American 
Institute  of  Aeronautics  and  Astronautics  48  (2010)  1513-1522. 


18 


Table  1:  Normalized  root-mean-square  difference  (RMSD)  (see  Eqn.3)  in  mean  and  stan¬ 
dard  deviation  statistics  obtained  using  the  reduced  representation  with  ISAT/RCCE 
relative  to  full  representation  with  ISAT  alone.  The  quantities  listed  are  temperature  T, 
density  p,  and  mass  fraction  of  species  CH 4,  O 2,  CO 2,  H20,  CO,  H2l  OH. 


Quantity 

£ 

Reference  Value 

£re/ 

RMSD  in  (0 
e((f))>  Percent 

RMSD  in  (^") 
e ((£")),  percent 

T(  K) 

1000.0 

2.92 

1.95 

p  (kg/m3) 

1.0 

1.72 

1.07 

YcHa 

1.5  x  10'1 

1.47 

1.17 

Yq2 

2.4  x  10'1 

2.10 

0.93 

YCo2 

1.2  x  10"1 

1.50 

0.81 

}'h20 

1.3  x  10"1 

2.24 

0.92 

YCO 

5.7  x  10"2 

6.36 

4.10 

YH2 

3.7  x  10"3 

7.87 

5.29 

Y OH 

2.7  x  10"3 

2.18 

1.92 

Table  2:  Cumulative  ISAT  statistics  for  the  LES/PDF  simulation  of  Sandia  Flame  D  using 
(i)  ISAT  alone  and  (ii)  ISAT/RCCE  with  the  P-URAN  parallel  strategy. 


method 

variables 

queries 

retrieves 

(percent) 

adds 

(percent) 

direct  evals^ 
(percent) 

retrieve 
time  (/xs) 

ISAT 

38 

3  X  1011 

99.974 

5.572  x  10“3 

1.665  x  10“3 

8 

ISAT/RCCE 

15 

3  x  1011 

99.984 

2.742  x  10“3 

0.0 

3 

(*) 


performed  only  if  the  ISAT  table  is  completely  filled. 
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Figure  1:  Radial  profiles  of  time-averaged  mean  temperature  T ,  density  p,  and  mass  frac¬ 
tion  of  species  CH^,  O2 ,  CO2,  H2O,  CO ,  H2,  OH  at  axial  locations  x/D  =  15,  30,  45  and 
60  obtained  from  (i)  experimental  data;  (ii)  an  LES/PDF  simulation  using  ISAT  alone  with 
the  38-species  full  representation;  and  (iii)  an  LES/PDF  simulation  using  ISAT/RCCE 
with  10  represented  species. 
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Figure  2:  Radial  profiles  of  time-averaged  standard  deviation  of  temperature  T,  density 
p,  and  mass  fraction  of  species  CH±,  02,  C02,  H20 ,  CO,  H2,  OH  at  axial  locations 
x/D  =  15,  30,  45  and  60  obtained  from  (i)  experimental  data;  (ii)  an  LES/PDF  simulation 
using  ISAT  alone  with  the  38-species  full  representation;  and  (iii)  an  LES/PDF  simulation 
using  ISAT/RCCE  with  10  represented  species. 
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Figure  3:  For  LES/PDF  simulation  of  Sandia  Flame  D,  wall  clock  time  for  2,000  time  steps 
along  with  breakdown  of  time  spent  in  LES,  HPDF  (outside  reaction),  Reaction  (including 
x2f.mpi  communication)  and  Waiting  (average  idle  time)  using  different  parallel  strategies. 
Top:  using  ISAT  alone  with  the  38-species  full  representation  with  the  P-URAN[0.2h,  32] 
parallel  strategy.  Bottom  three:  using  combined  ISAT/RCCE  with  10  represented  species 
using  (i)  PLP;  (ii)  URAN;  and  (iii)  P-URAN[0.2h,  32]  parallel  strategies. 
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